Using Group Membership Markers for Group Identification in Web Logs

We describe a system for automatically ranking documents by degree of militancy, designed as a tool both for finding militant websites and prioritizing the data found. Our ranking system employs a small hand-selected vocabulary based on group membership markers used by insiders to identify members and member properties (us) and outsiders and threats (them). We use the same vocabulary to build a classifier. Evaluating several ranking systems by their correlations with human judgments, we show that the best ranker uses the small us-them vocabulary, outperforming one system with a much larger vocabulary, and another with a small vocabulary chosen by Mutual Information. We confirm and extend recent results in sentiment analysis (Paltoglou and Thelwall 2010), showing that a featureweighting scheme taken from classical IR (TFIDF) produces the best ranking system; we also find, surprisingly, that adjusting these weights with SVM training, while producing a better classifier, produces a worse ranker. Increasing vocabulary size similarly improves classification (while worsening ranking). Finally, we experiment with adding usage models to both systems, models of how well each word’s syntactic usage pattern matches the usage pattern in a class model; this model does not benefit ranking, but increases the precision of the classifier. Our work complements and extends previous work tracking radical groups on the web (Chen 2007; Zhou et al. 2007; Burris, Smith, and Strahm 2000), which classified such sites with heterogeneous indicators, including document, vocabulary, and morphological features. The method combines elements of linguistics, machine learning, and behavioral science, and in principle can be extended to data collection aimed at any group organized for collective action.